Data processing method and apparatus

ABSTRACT

Embodiments of the present invention provide a data processing method and apparatus. According to the embodiments of the present invention, when it is found that a data hash value in a currently received data stream exceeds a preset first threshold, a part or all of data in the data stream is not deduplicated, and is directly stored, so as to prevent the data in the data stream from being dispersedly stored into a plurality of storage areas; instead, the part or all of the data is stored into a storage area in a centralized manner, so that a deduplication rate is effectively improved on the whole, particularly in a scenario of large data storage amount.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/CN2012/087879, filed on Dec. 28, 2012, which is hereby incorporatedby reference in its entirety.

TECHNICAL FIELD

Embodiments of the present invention relate to a storage technology, andin particular, to a data processing method and apparatus.

BACKGROUND

Data deduplication (briefly referred to as deduplication) is alsoreferred to as intelligent compression or single instance storage, andis a storage technology capable of automatically searching for duplicatedata, reserving only a unique copy of the same data, and replacing otherduplicate copies with a pointer that points to a single copy, so as toeliminate redundant data and reduce a storage capacity demand.

In a data deduplication solution in the prior art, received data ispartitioned to obtain data blocks, and then the data blocks form severaldata segments, an eigenvalue of each data segment is obtained throughcalculation by using a certain method, and a data segment is representedby an eigenvalue that is obtained through calculation. The eigenvalue ofthe data segment is matched with an eigenvalue of data stored in asystem, a storage area to which a storage address points is used as asimilar storage area, where the storage address corresponds to aneigenvalue in the system obtained through matching, data in the similarstorage area is loaded into a cache, and duplicate data query isperformed on the received data.

The inventor finds in research that, in existing data deduplication, forexample, data received for the first time is stored as new data; whendata received for the second time changes relative to the data receivedfor the first time, changing data is stored separately as new data; whendata received for the third time and the data received for the secondtime are the same, data that is the most similar to the data receivedfor the third time is probably still the data received for the firsttime; and in this way, relative to the data that changes for the firsttime, it is still considered that changing data is new data and isstored, while actually, the changing data has already been stored, andtherefore it can be seen that in deduplication processing in the priorart, the more the stored data is, the more the storage areas where thedata is dispersed are, and the whole deduplication performance isreduced.

SUMMARY

Embodiments of the present invention provide a data processing methodand apparatus, so as to effectively increase a deduplication rate of astorage system.

To achieve the inventive purpose, in a first aspect, an embodiment ofthe present invention provides a data processing method, including:

receiving a data stream, and acquiring eigenvalues that represent datain the data stream;

searching, according to a set index table, for a first storage addresscorresponding to each of the eigenvalues, where correspondence betweenan eigenvalue and a storage address where data represented by theeigenvalue is located is stored in the index table;

acquiring n second storage addresses from the first storage addressesaccording to a set policy, where n is greater than or equal to 1; and

when the number of the second storage addresses exceeds a set firstthreshold, directly regarding data in the received data stream as newdata, and storing the new data into a storage space.

In combination with the first aspect, in a first possible manner of thefirst aspect, the method further includes:

when the number of the second storage addresses does not exceed the setfirst threshold, comparing the data in the data stream with data instorage spaces to which the second storage addresses point, andsearching for duplicate data.

In combination with the first possible manner of the first aspect, in asecond possible manner, after the receiving the data stream, the methodfurther includes: segmenting the data in the data stream to obtain mdata segments, where m is an integer that is greater than 1.

The comparing the data in the data stream with data in storage spaces towhich the n second storage addresses point, and searching for duplicatedata includes:

comparing the data in the data stream with data in storage spaces towhich the n second storage addresses point; and for any one of the datasegments, if data in a data segment exists in storage spaces to which Sdifferent second storage addresses point, and the value of S exceeds aset second threshold, directly storing all the data in the data segmentinto a storage space as new data, where S is an integer that is greaterthan or equal to 1 and less than n.

In combination with the second possible manner of the first aspect, in athird possible manner, the comparing the data in the data stream withdata in storage spaces to which the second storage addresses point, andsearching for duplicate data further includes:

for any one of the data segments, if the data in the data segment existsin the storage spaces to which the S different second storage addressespoint, but the value of S does not exceed the set second threshold,regarding data in the data segment as new data, where the data is notfound through search in the storage spaces to which the n second storageaddresses point; and storing the new data into a storage space.

In combination with the first aspect or the first possible manner of thefirst aspect or the second possible manner of the first, in a fourthpossible manner, the storing the new data into a storage space includes:

storing the new data in a cache; and selecting a target storage addressused for writing data in the cache, and when a preset writing conditionis satisfied, writing the data in the cache into a storage space towhich the selected target storage address points, where the size of thewritten data and the size of the storage space to which the targetstorage address points are the same.

In combination with the fourth possible manner of the first aspect, in afifth possible manner, the method further includes: at the time ofwriting the data in the cache into a storage space to which the selectedtarget storage address points, recording data writing time of thestorage space into which the data is written.

The acquiring a similar second storage address from the first storageaddresses according to a set selection policy includes:

counting hits of the first storage addresses, and screening all the hitfirst storage addresses, where the screening includes: for the firststorage addresses with the same hits, according to recorded time atwhich data is written into storage spaces to which the first storageaddresses point, selecting the first storage address with latest time atwhich data is stored as an object used for selecting a similar secondstorage address; and selecting, according to the set selection policy, asimilar second storage address from the first storage addresses that areobtained after the screening.

In a second aspect, an embodiment of the present invention provides adata processing apparatus, including:

a receiving unit, configured to receive a data stream;

an eigenvalue acquiring unit, configured to acquire eigenvalues thatrepresent data in the data stream;

a first address acquiring unit, configured to search, according to a setindex table, for a first storage address corresponding to each of theeigenvalues, where correspondence between an eigenvalue and a storageaddress where data represented by the eigenvalue is located is stored inthe index table;

a second address acquiring unit, configured to acquire n second storageaddresses from the first storage addresses according to a set policy,where n is greater than or equal to 1;

a first determining unit, configured to: when it is determined that thenumber of the second storage addresses exceeds a set first threshold,directly regard data in the received data stream as new data; and

a storage unit, configured to store the new data into a storage space.

In combination with the first aspect, in the first possible manner, thefirst determining unit is further configured to: when it is determinedthat the number of the second storage addresses does not exceed the setfirst threshold, trigger a searching unit.

The searching unit is configured to compare the data in the data streamwith data in storage spaces to which the second storage addresses point,and search for duplicate data.

In combination with the first possible manner of the first aspect, inthe second possible manner, the apparatus further includes: a segmentingunit, configured to segment the data in the data stream to obtain m datasegments, where m is an integer that is greater than 1.

The searching unit includes:

a comparing sub-unit, configured to compare data in the data segmentswith the data in the storage spaces to which the n second storageaddresses point, determine, through search, whether the same dataexists, and send a searching result; and

a second determining sub-unit, configured to receive the searchingresult sent by the comparing unit; and for any one of the data segments,if data in a data segment exists in storage spaces to which S differentsecond storage addresses point, and the value of S exceeds a set secondthreshold, directly store all the data in the data segment into astorage space through the storage unit as new data, where S is aninteger that is greater than or equal to 1 and less than n.

In combination with the second possible manner of the first aspect, inthe third possible manner, the second determining sub-unit is furtherconfigured to: for any one of the data segments, if the data in the datasegment exists in the storage spaces to which the S different secondstorage addresses point, but the value of S does not exceed the setsecond threshold, regard data in the data segment as new data, where thedata is not found through search in the storage spaces to which the nsecond storage addresses point, and store the new data into a storagespace through the storage unit.

In combination with the first aspect or the first possible manner, thesecond possible manner, or the third possible manner of the firstaspect, in a fourth possible manner, the storage unit includes:

a cache sub-unit, configured to store the new data in a cache; and

a storage sub-unit, configured to select a target storage address usedfor writing data in the cache, and when a preset writing condition issatisfied, write the data in the cache into a storage space to which theselected target storage address points, where the size of the writtendata and the size of the storage space to which the target storageaddress points are the same.

In combination with the fourth possible manner of the first aspect, in afifth possible manner, the storage sub-unit is further configured to: atthe time of writing the data in the cache into a storage area to whichthe selected target storage address points, record data writing time ofthe storage area into which the data is written.

The second address acquiring unit is specifically configured to counthits of the first storage addresses, and screen all the hit firststorage addresses, where the screening includes: for the first storageaddresses with the same hits, according to recorded time at which datais written into storage spaces to which the first storage addressespoint, selecting the first storage address with latest time at whichdata is stored as an object used for selecting a similar second storageaddress; and select, according to a set selection policy, a similarsecond storage address from the first storage addresses that areobtained after the screening.

According to the embodiments of the present invention, when it is foundthat a data hash value in a currently received data stream exceeds apreset first threshold, a part or all of data in the data stream is notdeduplicated, and is directly stored, so as to prevent the data in thedata stream from being dispersedly stored into a plurality of storageareas. Because the data is aggregated, a data deduplication rate isapparently improved on the whole, particularly in a scenario of largedata storage amount.

BRIEF DESCRIPTION OF DRAWINGS

To describe the technical solutions in the embodiments of the presentinvention or in the prior art more clearly, the following brieflyintroduces the accompanying drawings required for describing theembodiments or the prior art. Apparently, the accompanying drawings inthe following description show merely some embodiments of the presentinvention, and persons of ordinary skill in the art may still deriveother drawings from these accompanying drawings without creativeefforts.

FIG. 1 is a flow chart of a data processing method according to anembodiment of the present invention;

FIG. 2 is a schematic diagram of an internal structure of a physicalnode according to an embodiment of the present invention;

FIG. 3 is a structural diagram of a data processing apparatus accordingto an embodiment of the present invention; and

FIG. 4 is a structural diagram of another data processing apparatusaccording to an embodiment of the present invention.

DESCRIPTION OF EMBODIMENTS

To make the objectives, technical solutions, and advantages of theembodiments of the present invention more comprehensible, the followingclearly and completely describes the technical solutions in theembodiments of the present invention with reference to the accompanyingdrawings in the embodiments of the present invention. Apparently, theembodiments to be described are merely a part rather than all of theembodiments of the present invention. All other embodiments obtained bypersons of ordinary skill in the art based on the embodiments of thepresent invention without creative efforts shall fall within theprotection scope of the present invention.

The embodiments of the present invention may be applied to a storagesystem, the storage system may include a plurality of physical nodes,and may also include only one physical node, which is not limited in theembodiments of the present invention. A physical node having adeduplication engine may be used as an executing subject of theembodiments of the present invention, and execute a method in anembodiment of the present invention after receiving a deduplicationtask.

FIG. 1 is a flow chart of a data processing method according to anembodiment of the present invention. As shown in FIG. 1, the method mayinclude:

Step 10: Receive a data stream.

Step 11: Acquire eigenvalues that represent data in the data stream.

A method for acquiring the eigenvalues of the data in the received datastream may be obtained in many manners. For example, the data is dividedinto data blocks, a plurality of data blocks forms one data segment,thereby obtaining a plurality of data segments, and a minimum hash valueis extracted from hash values of data blocks in each data segment as aneigenvalue of a data segment where the minimum hash value belongs. Inaddition, eigenvalues of a data stream may be further obtained in manymanners, reference may be made to the prior art, which is not limited inthe embodiment of the present invention.

Step 12: Search, according to a set index table, for a first storageaddress corresponding to each of the eigenvalues, where correspondencebetween an eigenvalue and a storage address where data represented bythe eigenvalue is located is stored in the index table.

Data blocks and fingerprint information corresponding to the data blocksare stored in storage areas to which different storage addresses point.In specific implementation, a storage area to which one storage addresspoints may have a plurality of groups of data, and if one eigenvalue isselected in each group, a case that one storage address corresponds to aplurality of different eigenvalues occurs, and therefore the samestorage address in an index table may correspond to a plurality ofdifferent eigenvalues, but the same eigenvalue corresponds to onestorage address. In the embodiment of the present invention, a storagearea to which a storage address points and a storage space to which astorage address points have the same meaning, but only are differentexpression manners.

Step 13: Acquire n second storage addresses from the first storageaddresses according to a set policy, where n is greater than or equal to1.

A similar second storage address means that data stored in a storagearea to which a second storage address points is similar to the data inthe received data stream, and possibly, duplicate data is much.

The index table is stored in a memory in the storage according to a setpolicy, and data blocks and fingerprint information corresponding to thedata blocks are stored in storage areas to which different storageaddresses point. A storage area corresponding to one storage address hasseveral pieces of data, and if a plurality of eigenvalues is selectedfrom the data in the storage area, a case that one storage addresscorresponds to a plurality of different eigenvalues occurs, andtherefore the same storage address in an index table may correspond to aplurality of different eigenvalues, but the same eigenvalue correspondsto one storage address. When a plurality of eigenvalues of the receiveddata stream is queried in an index table, a plurality of correspondingfirst storage addresses may be obtained, and the first storage addresscorresponding to an eigenvalue of the received data stream is referredto as a hit first storage address.

There may be a plurality of policies for selecting a second storageaddress from the obtained plurality of first storage addresses, and thepolicies are set by a user, for example:

A first storage address, hits of which exceed a preset third threshold,is selected from the first storage addresses as a similar second storageaddress; or, all different hit first storage addresses are regarded assecond storage addresses; or, hits of different first storage addressesare counted, the hits are sequenced in descending order, and differentfirst storage addresses with the same hits are sequenced with the sameserial number; and then first storage addresses with previous N serialnumbers are selected. For example, hits of a storage address 1 is 3,hits of a storage address 2 is 4, and hits of a storage address 3 isalso 4, and when the storage addresses 1, 2 and 3 are sequenced, serialnumbers of the storage addresses 2 and 3 both are 2. If a preset policyis to select previous two first storage addresses as second storageaddresses, the number of second storage addresses is three, and thesecond storage addresses include: the storage addresses 1, 2 and 3.

Step 14: When the number of the second storage addresses exceeds a setfirst threshold, directly regard the data in the received data stream asnew data, and store the new data into a storage space.

The new data is data that is not stored in a storage system; anddefinitely, in specific implementation, the new data is considered by anexecuting subject, in a duplicate data searching process, as data thatis not stored in the storage system, and is not objectively consideredas data that definitely does not exist in the storage system.

The user sets a first threshold, and when the number of the secondstorage addresses exceeds the first threshold, it means that the data inthe received data stream possibly exists dispersedly in the secondstorage addresses the number of which exceeds the first threshold, andtherefore the first threshold may also be referred to as a hash value ofa data stream. In this case, if the received data stream furtherincludes new data, the new data may be re-stored in a storage area towhich another storage address except the second storage addressespoints; while in the embodiment of the present invention, in this case,the data in the received data stream is regarded as new data and isstored, so as to prevent the data in the received data stream from beingdispersedly stored into storage areas to which a plurality of storageaddresses points.

In the embodiment of the present invention, when the number of thesecond storage addresses exceeds the preset first threshold, a part orall of the data in the received data stream is regarded as new data, theamount of data of received data that specifically needs to be used asnew data may be set by the user according to an actual situation, forexample, set according to a data percentage, which is not limited in theembodiment of the present invention.

For an exemplary schematic diagram of an internal structure of aphysical node in an embodiment of the invention, reference may be madeto FIG. 2. A physical node executing a deduplication task furtherincludes a storage apparatus, which enables each physical node to savedata for a long time, and the storage apparatus may be a disk, and mayalso be another storage apparatus, such as SSD, and a storage apparatuson each physical node is referred to as a single instance repository(single instance repository, SIR). A storage apparatus of a physicalnode has many storage areas. In a redundant arrays of inexpensive disks(Redundant Arrays of Inexpensive Disks, RAID) system, one storage areamay be regarded as one stripe, each storage area may be visuallyconsidered as one container (container) that stores data in specificimplementation, each storage container has one number, which may bereferred to as a storage container identity (container ID, CID), andthis container identity indicates a position of this storage containerin a storage system, for example: indicates that this storage containeris in which storage area on which physical node in the storage system.Then, a storage address of a stored data block mentioned above ispresented as a CID in specific implementation, indicating that the datablock is stored in which storage area on which physical node, and for aneigenvalue, the aforementioned correspondence between an eigenvalue inan index table and a storage address of a stored data block representedby the eigenvalue may be embodied as correspondence between aneigenvalue and a CID in specific implementation; and in addition to adata block, fingerprint information corresponding to the data block mayfurther be stored in each storage area.

A storage apparatus of a physical node has many storage areas. Eachstorage area may be visually considered as one container (container)that stores data in specific implementation, each storage container hasone number, which may be referred to as a storage container identity(container ID, CID), and this container identity indicates a position ofthis storage container in a storage system, for example: indicates thatthis storage container is in which storage area on which physical nodein the storage system. Then, a storage address of a stored data blockmentioned above is presented as a CID in specific implementation,indicating that the data block is stored in which storage area on whichphysical node, and the aforementioned correspondence between aneigenvalue in an index table and a storage address of a stored datablock represented by the eigenvalue may be embodied as correspondencebetween an eigenvalue and a CID in specific implementation; and inaddition to a data block, fingerprint information corresponding to thedata block may further be stored in each storage area.

Data in a container buffer in a cache where the new data is stored iswholly written into a container of a storage apparatus of a physicalnode, and the size of each storage area in a cache for storing data andthe size of each storage area on a target physical node to which data ismigrated are the same, that is, the size of each container buffer andthe size of each container are the same; generally only after onecontainer is fully stored with data, data can be written into a newcontainer. A storage area in a cache of a current physical node is usedfor temporarily storing new data that is found through search in a datadeduplication process, that is to say, data in one storage area in thecache includes data that is considered by the current physical node asnew data in a duplicate data searching process, no matter whethermethods for acquiring the new data are the same.

Therefore, the regarding a part or all of the data in the received datastream as new data and storing the new data into a storage space may beimplemented through the following method.

The part or all of the data in the received data stream is regarded asnew data and stored in a cache; and a target storage address used forwriting data in the cache is selected, and when a preset writingcondition is satisfied, the data in the cache is written into a storagearea to which the selected target storage address points, where the sizeof the written data and the size of the storage area to which the targetstorage address points are the same.

In specific implementation, a cache has at least one container buffer,and when one container buffer is fully stored with data, the data in thecontainer buffer may be written into a container corresponding to astorage address that is selected in a storage apparatus.

Step 15: Insert correspondence between an eigenvalue that represents thenew data and a storage address of the new data into the index table.

An index table is stored on a physical node, and correspondence betweenan eigenvalue and a storage address of a stored data block representedby the eigenvalue is stored in the index table.

It can be seen from the foregoing embodiment that, when it is found thata data hash value in a currently received data stream exceeds a presetfirst threshold, a part or all of data in the data stream is notdeduplicated, and is directly stored, so as to prevent the data in thedata stream from being dispersedly stored into a plurality of storageareas, and a deduplication rate in current data deduplication isreduced; but the received data stream is not lost, and is stored into astorage area in a centralized manner, and the deduplication rate isimproved in next data deduplication, and therefore a data deduplicationrate is apparently improved on the whole, particularly in a scenario oflarge data storage amount. For example, data received for the first timeis 123; after the data is stored as new data, data received for thesecond time is 124, 4 is separately stored in one storage area as newdata in the prior art, and when the data 124 is received for the thirdtime, a most similar storage area is still an area for storing the data123, and then 4 is still used as new data; while in the solution in theembodiment of the present invention, when a certain condition issatisfied, the data 124 received for the second time is directly storedin one storage area as new data, and when the data 124 is received forthe third time, it is found, through search, that the most similarstorage area includes 124, and therefore 4 is not stored as new dataagain.

If the number of the second storage addresses does not exceed the firstthreshold, the implementation of the present invention further includes:

Step 16: When the number of the second storage addresses does not exceedthe set first threshold, compare the data in the data stream with datain storage spaces to which the second storage addresses point, andsearch for duplicate data.

After the receiving the data stream in step 10 in the embodiment of thepresent invention, the following step may further be included.

Step 10 a: Segment the received data stream to obtain m data segments,where m is an integer that is greater than 1.

Correspondingly, the comparing the data in the data stream with data instorage spaces to which the n second storage addresses point, andsearching for duplicate data in step 16 includes:

comparing the data in the data stream with data in storage spaces towhich the n second storage addresses point; and for any one of the datasegments, if data in a data segment exists in storage spaces to which Sdifferent second storage addresses point, and the value of S exceeds aset second threshold, directly storing all the data in the data segmentinto a storage space as new data; and skipping to step 15, where S is aninteger that is greater than or equal to 1 and less than n.

By skipping to step 15, correspondence between an eigenvalue of a datasegment that satisfies a condition and a storage address of data in thedata segment that is obtained through determination is inserted into theindex table.

The comparing the data in the data stream with data in storage spaces towhich the n second storage addresses point, and searching for duplicatedata in step 16 may further include:

for any one of the data segments, if the data in the data segment existsin the storage spaces to which the S different second storage addressespoint, but the value of S does not exceed the set second threshold,regarding data in the data segment as new data, where the data is notfound through search in the storage spaces to which the n second storageaddresses point; and storing the new data into a storage space; andskipping to step 15.

By skipping to step 15, correspondence between new data in a datasegment and a storage address of the new data in the data segment isinserted into the index table.

In the embodiment of the present invention, in a duplicate datasearching process, in addition to determining a hash value of a datastream, a hash value of a data segment is further determined, and whenit is found that data in the data segment exists in a storage areaexcessively dispersedly, the data in the data segment is regarded as newdata for processing, thereby better aggregating the data, so as to moreprecisely determine, during subsequent deduplication, whether the datais duplicate data, and improve a deduplication rate.

In the embodiment of the present invention, at the time of selecting asecond storage address, first storage addresses used as objects forselecting second storage addresses may be screened, and then a similarsecond storage address is selected, according to a set policy, from thefirst storage addresses that are obtained after the screening, andtherefore the embodiment of the present invention further includes:

at the time of writing the data in the cache into a storage space towhich the selected target storage address points, recording data writingtime of the storage space into which the data is written.

Correspondingly, the acquiring a similar second storage address from thefirst storage addresses according to a set selection policy in step 13in the embodiment of the present invention may include:

counting hits of the first storage addresses, and screening all the hitfirst storage addresses, where the screening includes: for the firststorage addresses with the same hits, according to recorded time atwhich data is written into spaces corresponding to the first storageaddresses, selecting the first storage address with latest time at whichdata is stored as an object used for selecting a similar second storageaddress; and selecting, according to the set selection policy, a similarsecond storage address from the first storage addresses that areobtained after the screening.

In specific implementation, a storage area with latest data writing timemeans that data of the area is relatively new, and if it isdistinguished according to coldness and hotness of data, data withlatest writing time is probably hotter, and therefore among the firststorage addresses with the same hits, a storage address with latest datawriting time is selected preferably. For example, hits of a firststorage address 1 is five times, hits of a first storage address 2 isthree times, hits of a first storage address 3 is three times, hits of afirst storage address 4 is three times, hits of a first storage address5 is twice, and then according to the method in the embodiment of thepresent invention, the first storage addresses with the hits that arethree times are screened first. If data storing time of the firststorage address 3 is the latest, objects used for selecting a secondstorage address after the screening include only: the first storageaddress 1, the first storage address 3, and the first storage address 5,and then, a similar second storage address is selected from the firststorage addresses 1, 3, and 5 according to a set selection policy.

In the embodiment of the present invention, when it is found that a datahash value in a currently received data stream exceeds a preset firstthreshold, a part or all of data in the data stream is not deduplicated,and is directly stored, so as to aggregate excessively dispersed data ina storage apparatus, and improve a deduplication rate on the whole,particularly in a case of mass data storage.

An embodiment of the present invention further provides a dataprocessing apparatus, which is applicable to a storage system, disposedin a physical node in the storage system, and configured to execute thedata processing method described in the foregoing method embodiment, andduring specific implementation, the data processing apparatus may be adeduplication engine.

Referring to FIG. 3, the data processing apparatus provided in theembodiment of the present invention may include:

a receiving unit 30, configured to receive a data stream;

an eigenvalue acquiring unit 31, configured to acquire eigenvalues thatrepresent data in the data stream,

where the eigenvalue acquiring unit 31 acquires the eigenvalues of thedata in the received data stream in a plurality of manners duringspecific implementation, and reference may be made to the description inthe method embodiment;

a first address acquiring unit 32, configured to search, according to aset index table, for a first storage address corresponding to each ofthe eigenvalues, where correspondence between an eigenvalue and astorage address where data represented by the eigenvalue is located isstored in the index table; and

a second address acquiring unit 33, configured to acquire n secondstorage addresses from the first storage addresses according to a setpolicy, where n is greater than or equal to 1.

A similar second storage address means that data stored in a storagearea to which a second storage address points is similar to the data inthe received data stream, and possibly, duplicate data is much.

The index table is stored in a memory in the storage according to a setpolicy, and data blocks and fingerprint information corresponding to thedata blocks are stored in storage areas to which different storageaddresses point. A storage area corresponding to one storage address hasseveral pieces of data, and if a plurality of eigenvalues is selectedfrom the data in the storage area, a case that one storage addresscorresponds to a plurality of different eigenvalues occurs, andtherefore the same storage address in an index table may correspond to aplurality of different eigenvalues, but the same eigenvalue correspondsto one storage address. When a plurality of eigenvalues of the receiveddata stream is queried in an index table, a plurality of correspondingfirst storage addresses may be obtained, and the first storage addresscorresponding to an eigenvalue of the received data stream is referredto as a hit first storage address.

There may be a plurality of policies for selecting a second storageaddress from the obtained plurality of first storage addresses, which isnot limited in the embodiment of the present invention.

A first determining unit 34 is configured to: when it is determined thatthe number of the second storage addresses exceeds a set firstthreshold, directly regard data in the received data stream as new data.

In the embodiment of the present invention, the first determining unit34 is specifically configured to: when the number of the second storageaddresses exceeds the preset first threshold, regard a part or all ofthe data in the received data stream as new data.

The amount of data of received data that specifically needs to be usedas new data may be set by a user according to an actual situation, forexample, set according to a data percentage, which is not limited in theembodiment of the present invention.

A storage unit 35 is configured to store the new data into a storagespace.

Optionally, the storage unit 35 includes:

a cache sub-unit 351, configured to store the new data in a cache; and

a storage sub-unit 352, configured to select a target storage addressused for writing data in the cache, and when a preset writing conditionis satisfied, write the data in the cache into a storage space to whichthe selected target storage address points, where the size of thewritten data and the size of the storage space to which the targetstorage address points are the same.

Optionally, the storage sub-unit 352 is further configured to: at thetime of writing the data in the cache into a storage area to which theselected target storage address points, record data writing time of thestorage area into which the data is written.

On such a basis, the second address acquiring unit 33 is specificallyconfigured to count hits of the first storage addresses, and screen allthe hit first storage addresses, where the screening includes: for thefirst storage addresses with the same hits, according to recorded timeat which data is written into storage spaces to which the first storageaddresses point, selecting the first storage address with latest time atwhich data is stored as an object used for selecting a similar secondstorage address; and select, according to a set selection policy, asimilar second storage address from the first storage addresses that areobtained after the screening.

With the apparatus provided in the embodiment of the present invention,when a first determining unit finds that a data hash value in acurrently received data stream exceeds a preset first threshold, data inthe data stream is not deduplicated, and is directly regarded as newdata, and the new data is stored by a storage unit, so as to prevent thedata in the data stream from being dispersedly stored into a pluralityof storage areas, so that a deduplication rate in current datadeduplication is reduced; but the received data stream is not lost, andis stored into a storage area in a centralized manner, and thededuplication rate is improved in next data deduplication, and thereforea data deduplication rate is apparently improved on the whole,particularly in a scenario of large data storage amount.

Optionally, the data processing apparatus provided in the embodiment ofthe present invention may further include a searching unit 36.

The first determining unit 34 is further configured to: when it isdetermined that the number of the second storage addresses does notexceed the set first threshold, trigger the searching unit.

The searching unit 36 is configured to compare the data in the datastream with data in storage spaces to which the second storage addressespoint, and search for duplicate data.

Optionally, the embodiment of the present invention may further include:

a segmenting unit 31 a, configured to segment the data in the datastream to obtain m data segments, where m is an integer that is greaterthan 1.

The searching unit 36 may determine, by using a data segment as a unit,whether data in a data segment is excessively dispersed, at the time ofperforming duplicate data searching on data, and therefore in a casethat the data processing apparatus further includes the segmenting unit31 a, the searching unit 36 may include:

a comparing sub-unit 361, configured to compare data in the datasegments with the data in the storage spaces to which the n secondstorage addresses point, determine, through search, whether the samedata exists, and send a searching result; and

a second determining sub-unit 362, configured to receive the searchingresult sent by the comparing unit; and for any one of the data segments,if data in a data segment exists in storage spaces to which S differentsecond storage addresses point, and the value of S exceeds a set secondthreshold, directly store all the data in the data segment into astorage space through the storage unit as new data, where S is aninteger greater that is than or equal to 1 and less than n.

Optionally, the second determining unit 362 may further be configuredto: for any one of the data segments, if the data in the data segmentexists in the storage spaces to which the S different second storageaddresses point, but the value of S does not exceed the set secondthreshold, regard data in the data segment as new data, where the datais not found through search in the storage spaces to which the n secondstorage addresses point, and store the new data into a storage spacethrough the storage unit.

Optionally, the embodiment of the present invention may further include:

an index updating unit 37, configured to insert correspondence betweenan eigenvalue that represents the new data and a storage address of thenew data into the index table.

With the data processing apparatus provided in the present invention, ina case that data in a data stream is excessively dispersed in a storagesystem, data may not be deduplicated, and is directly stored; and a datasegment in the data stream may be determined, and in a case that data inthe data segment is excessively dispersed, the data in the data segmentis not deduplicated, thereby effectively preventing the data in the datastream from being dispersed into too many storage areas, so that adeduplication rate is improved on the whole.

Referring to FIG. 4, an embodiment of the present invention furtherprovides a data processing apparatus 400, including: a processor 40, amemory 41, a bus 42, and a communication interface 43, where theprocessor 40, the communication interface 43, and the memory 41 areconnected through the bus 42.

The memory 41 is configured to store a program 401.

The processor 40 is configured to execute the program 401 in the memory41, where the processor 40 receives a data stream through thecommunication interface 43.

In specific implementation, the program 401 may include a program code,where the program code includes a computer operating instruction.

The processor 102 may be a central processing unit CPU, or anapplication specific integrated circuit ASIC (Application SpecificIntegrated Circuit), or be configured as one or more integrated circuitsimplementing the embodiments of the present invention.

Referring to FIG. 3, the program 401 may include:

a receiving unit 30, configured to receive a data stream;

an eigenvalue acquiring unit 31, configured to acquire eigenvalues thatrepresent data in the data stream,

where the eigenvalue acquiring unit 31 acquires the eigenvalues of thedata in the received data stream in a plurality of manners duringspecific implementation, and reference may be made to the description inthe method embodiment;

a first address acquiring unit 32, configured to search, according to aset index table, for a first storage address corresponding to each ofthe eigenvalues, where correspondence between an eigenvalue and astorage address where data represented by the eigenvalue is located isstored in the index table; and

a second address acquiring unit 33, configured to acquire n secondstorage addresses from the first storage addresses according to a setpolicy, where n is greater than or equal to 1.

A similar second storage address means that data stored in a storagearea to which a second storage address points is similar to the data inthe received data stream, and possibly, duplicate data is much.

The index table is stored in a memory in the storage according to a setpolicy, and data blocks and fingerprint information corresponding to thedata blocks are stored in storage areas to which different storageaddresses point. A storage area corresponding to one storage address hasseveral pieces of data, and if a plurality of eigenvalues is selectedfrom the data in the storage area, a case that one storage addresscorresponds to a plurality of different eigenvalues occurs, andtherefore the same storage address in an index table may correspond to aplurality of different eigenvalues, but the same eigenvalue correspondsto one storage address. When a plurality of eigenvalues of the receiveddata stream is queried in an index table, a plurality of correspondingfirst storage addresses may be obtained, and the first storage addresscorresponding to an eigenvalue of the received data stream is referredto as a hit first storage address.

There may be a plurality of policies for selecting a second storageaddress from the obtained plurality of first storage addresses, which isnot limited in the embodiment of the present invention.

A first determining unit 34 is configured to: when it is determined thatthe number of the second storage addresses exceeds a set firstthreshold, directly regard data in the received data stream as new data.

In the embodiment of the present invention, the first determining unit34 is specifically configured to: when the number of the second storageaddresses exceeds the preset first threshold, regard a part or all ofthe data in the received data stream as new data.

The amount of data of received data that specifically needs to be usedas new data may be set by a user according to an actual situation, forexample, set according to a data percentage, which is not limited in theembodiment of the present invention.

A storage unit 35 is configured to store the new data into a storagespace.

Optionally, the storage unit 35 includes:

a cache sub-unit 351, configured to store the new data in a cache; and

a storage sub-unit 352, configured to select a target storage addressused for writing data in the cache, and when a preset writing conditionis satisfied, write the data in the cache into a storage space to whichthe selected target storage address points, where the size of thewritten data and the size of the storage space to which the targetstorage address points are the same.

Optionally, the storage sub-unit 352 is further configured to: at thetime of writing the data in the cache into a storage area to which theselected target storage address points, record data writing time of thestorage area into which the data is written.

On such a basis, the second address acquiring unit 33 is specificallyconfigured to count hits of the first storage addresses, and screen allthe hit first storage addresses, where the screening includes: for thefirst storage addresses with the same hits, according to recorded timeat which data is written into storage spaces to which the first storageaddresses point, selecting the first storage address with latest time atwhich data is stored as an object used for selecting a similar secondstorage address; and select, according to a set selection policy, asimilar second storage address from the first storage addresses that areobtained after the screening.

Optionally, the data processing apparatus provided in the embodiment ofthe present invention may further include a searching unit 36.

The first determining unit 34 is further configured to: when it isdetermined that the number of the second storage addresses does notexceed the set first threshold, trigger the searching unit.

The searching unit 36 is configured to compare the data in the datastream with data in storage spaces to which the second storage addressespoint, and search for duplicate data.

Optionally, the embodiment of the present invention may further include:

a segmenting unit 31 a, configured to segment the data in the datastream to obtain m data segments, where m is an integer that is greaterthan 1.

The searching unit 36 may determine, by using a data segment as a unit,whether data in a data segment is excessively dispersed, at the time ofperforming duplicate data searching on data, and therefore in a casethat the data processing apparatus further includes the segmenting unit31 a, the searching unit 36 may include:

a comparing sub-unit 361, configured to compare data in the datasegments with the data in the storage spaces to which the n secondstorage addresses point, determine, through search, whether the samedata exists, and send a searching result; and

a second determining sub-unit 362, configured to receive the searchingresult sent by the comparing unit; and for any one of the data segments,if data in a data segment exists in storage spaces to which S differentsecond storage addresses point, and the value of S exceeds a set secondthreshold, directly store all the data in the data segment into astorage space through the storage unit as new data, where S is aninteger that is greater than or equal to 1 and less than n.

Optionally, the second determining unit 362 may further be configuredto: for any one of the data segments, if the data in the data segmentexists in the storage spaces to which the S different second storageaddresses point, but the value of S does not exceed the set secondthreshold, regard data in the data segment as new data, where the datais not found through search in the storage spaces to which the n secondstorage addresses point, and store the new data into a storage spacethrough the storage unit.

Optionally, the embodiment of the present invention may further include:

an index updating unit 37, configured to insert correspondence betweenan eigenvalue that represents the new data and a storage address of thenew data into the index table.

With the data processing apparatus provided in the present invention, ina case that data in a data stream is excessively dispersed in a storagesystem, data may not be deduplicated, and is directly stored; and a datasegment in the data stream may be determined, and in a case that data inthe data segment is excessively dispersed, the data in the data segmentis not deduplicated, thereby effectively preventing the data in the datastream from being dispersed into too many storage areas, so that adeduplication rate is improved on the whole.

A computer program product for executing data processing provided in theembodiment of the present invention includes a computer readable storagemedium storing a program code, an instruction included in the programcode may be used for executing the method in the foregoing methodembodiment, and for specific implementation, reference may be made tothe method embodiment, which is not described herein again.

It may be clearly understood by persons skilled in the art that, for thepurpose of convenient and brief description, for a detailed workingprocess of the foregoing system, apparatus, and unit, reference may bemade to a corresponding process in the foregoing method embodiment,which is not described herein again.

In the several embodiments provided in the present application, itshould be understood that the disclosed system, apparatus, and methodmay be implemented in other manners. For example, the foregoingdescribed apparatus embodiment is merely exemplary. For example, theunit division is merely logical function division and may be otherdivision in actual implementation. For example, a plurality of units orcomponents may be combined or integrated into another system, or somefeatures may be ignored or not performed. In addition, the displayed ordiscussed mutual couplings or direct couplings or communicationconnections may be implemented through some interfaces. The indirectcouplings or communication connections between the apparatuses or unitsmay be implemented in electronic, mechanical or other forms.

The units described as separate parts may or may not be physicallyseparate, and parts displayed as units may or may not be physical units,may be located in one position, or may be distributed on a plurality ofnetwork units. A part or all of the units may be selected according toan actual need to achieve the objectives of the solutions of theembodiments.

In addition, functional units in the embodiments of the presentinvention may be integrated into one processing unit, or each of theunits may exist alone physically, or two or more units are integratedinto one unit.

When the functions are implemented in the form of a software functionalunit and sold or used as an independent product, the functions may bestored in a computer-readable storage medium. Based on such anunderstanding, the technical solutions of the present inventionessentially, or the part contributing to the prior art, or part of thetechnical solutions may be implemented in the form of a softwareproduct. The computer software product is stored in a storage medium,and includes several instructions for instructing a computer device(which may be a personal computer, a server, or a network device) toperform all or a part of the steps of the method described in theembodiments of the present invention. The foregoing storage mediumincludes: any medium that can store program codes, such as a USB flashdisk, a removable hard disk, a read-only memory (ROM, Read-Only Memory),a random access memory (RAM, Random Access Memory), a magnetic disk, oran optical disk.

The foregoing descriptions are merely specific embodiments of thepresent invention, but are not intended to limit the protection scope ofthe present invention. Any variation or replacement readily figured outby persons skilled in the art within the technical scope disclosed inthe present invention shall all fall within the protection scope of thepresent invention. Therefore, the protection scope of the presentinvention shall be subject to the protection scope of the claims.

What is claimed is:
 1. A data processing method, comprising: receiving adata stream, and acquiring eigenvalues that represent data in the datastream; searching, according to a set index table, for a first storageaddress corresponding to each of the eigenvalues, wherein correspondencebetween an eigenvalue and a storage address where data represented bythe eigenvalue is located is stored in the index table; acquiring nsecond storage addresses from the first storage addresses according to aset policy, wherein n is greater than or equal to 1; and when the numberof the second storage addresses exceeds a set first threshold, directlyregarding data in the received data stream as new data, and storing thenew data into a storage space.
 2. The method according to claim 1,further comprising: when the number of the second storage addresses doesnot exceed the set first threshold, comparing the data in the datastream with data in storage spaces to which the second storage addressespoint, and searching for duplicate data.
 3. The method according toclaim 2, wherein after the receiving the data stream, the method furthercomprises: segmenting the data in the data stream to obtain m datasegments, wherein m is an integer that is greater than 1; and thecomparing the data in the data stream with data in storage spaces towhich the n second storage addresses point, and searching for duplicatedata comprises: comparing the data in the data stream with data instorage spaces to which the n second storage addresses point; and forany one of the data segments, if data in a data segment exists instorage spaces to which S different second storage addresses point, andthe value of S exceeds a set second threshold, directly storing all thedata in the data segment into a storage space as new data, wherein S isan integer that is greater than or equal to 1 and less than n.
 4. Themethod according to claim 3, wherein the comparing the data in the datastream with data in storage spaces to which the second storage addressespoint, and searching for duplicate data further comprises: for any oneof the data segments, if the data in the data segment exists in thestorage spaces to which the S different second storage addresses point,but the value of S does not exceed the set second threshold, regardingdata in the data segment as new data, wherein the data is not foundthrough search in the storage spaces to which the n second storageaddresses point; and storing the new data into a storage space.
 5. Themethod according to any one of claim 1, wherein the directly regardingthe data in the received data stream as new data and storing the newdata into a storage space comprises: directly regarding a part or all ofthe data in the received data stream as new data and storing the newdata into a storage space.
 6. The method according to any one of claim1, wherein the storing the new data into a storage space comprises:storing the new data in a cache; and selecting a target storage address,and when a preset writing condition is satisfied, writing the data inthe cache into a storage space to which the selected target storageaddress points, wherein the size of the written data and the size of thestorage space to which the target storage address points are the same.7. The method according to claim 6, wherein the method furthercomprises: at the time of writing the data in the cache into a storagespace to which the selected target storage address points, recordingdata writing time of the storage space into which the data is written;and the acquiring n second storage addresses from the first storageaddresses according to a set selection policy comprises: counting hitsof the first storage addresses, and screening all the hit first storageaddresses, wherein the screening comprises: for the first storageaddresses with the same hits, according to recorded time at which thedata is written into storage spaces to which the first storage addressespoint, selecting the first storage address with latest time at whichdata is stored as an object used for selecting a second storage address;and selecting, according to the set selection policy, a second storageaddress from the first storage addresses that are obtained after thescreening.
 8. The method according to any one of claim 1, furthercomprising: inserting correspondence between an eigenvalue thatrepresents the new data and a storage address of the new data into theindex table.
 9. A data processing apparatus in a cluster system,comprising: a processor, a memory, a communication interface, and a bus,wherein the processor, the communication interface, and the memorycommunicate with each other through the bus; the memory is configured tostore a program; and wherein the processor receives a data streamthrough the communication interface and the memory is configured toprovide the processor with instructions for: receiving a data stream,and acquiring eigenvalues that represent data in the data stream;searching, according to a set index table, for a first storage addresscorresponding to each of the eigenvalues, wherein correspondence betweenan eigenvalue and a storage address where data represented by theeigenvalue is located is stored in the index table; acquiring n secondstorage addresses from the first storage addresses according to a setpolicy, wherein n is greater than or equal to 1; and when the number ofthe second storage addresses exceeds a set first threshold, directlyregarding data in the received data stream as new data, and storing thenew data into a storage space.
 10. The data processing apparatusaccording to claim 9, wherein the memory is further configured toprovide the processor with instructions for: when the number of thesecond storage addresses does not exceed the set first threshold,comparing the data in the data stream with data in storage spaces towhich the second storage addresses point, and searching for duplicatedata.
 11. The data processing apparatus according to claim 10, whereinafter the receiving the data stream, the memory is further configured toprovide the processor with instructions for: segmenting the data in thedata stream to obtain m data segments, wherein m is an integer that isgreater than 1; and the comparing the data in the data stream with datain storage spaces to which the n second storage addresses point, andsearching for duplicate data comprises: comparing the data in the datastream with data in storage spaces to which the n second storageaddresses point; and for any one of the data segments, if data in a datasegment exists in storage spaces to which S different second storageaddresses point, and the value of S exceeds a set second threshold,directly storing all the data in the data segment into a storage spaceas new data, wherein S is an integer that is greater than or equal to 1and less than n.
 12. The data processing apparatus according to claim11, wherein the comparing the data in the data stream with data instorage spaces to which the second storage addresses point, andsearching for duplicate data further comprises: for any one of the datasegments, if the data in the data segment exists in the storage spacesto which the S different second storage addresses point, but the valueof S does not exceed the set second threshold, regarding data in thedata segment as new data, wherein the data is not found through searchin the storage spaces to which the n second storage addresses point; andstoring the new data into a storage space.
 13. The data processingapparatus according to claim 9, wherein the directly regarding the datain the received data stream as new data and storing the new data into astorage space comprises: directly regarding a part or all of the data inthe received data stream as new data and storing the new data into astorage space.
 14. The data processing apparatus according to claim 9,wherein the storing the new data into a storage space comprises: storingthe new data in a cache; and selecting a target storage address, andwhen a preset writing condition is satisfied, writing the data in thecache into a storage space to which the selected target storage addresspoints, wherein the size of the written data and the size of the storagespace to which the target storage address points are the same.
 15. Thedata processing apparatus according to claim 14, wherein the memory isfurther configured to provide the processor with instructions for: atthe time of writing the data in the cache into a storage space to whichthe selected target storage address points, recording data writing timeof the storage space into which the data is written; and the acquiring asecond storage address from the first storage addresses according to aset selection policy comprises: counting hits of the first storageaddresses, and screening all the hit first storage addresses, whereinthe screening comprises: for the first storage addresses with the samehits, according to recorded time at which the data is written intostorage spaces to which the first storage addresses point, selecting thefirst storage address with latest time at which data is stored as anobject used for selecting a second storage address; and selecting,according to the set selection policy, a second storage address from thefirst storage addresses that are obtained after the screening.
 16. Acomputer program product for executing data processing, comprising acomputer readable storage medium storing a program code, wherein aninstruction comprised in the program code is used for executing themethod according to any one of claims 1 to 8.